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Creating  an  Online  Laboratory 
David  Lazer,  Waleed  Melies,  Jason  Radford 

Statement  of  the  Problem  Studied 

Performing  experimental  research  on  social  networks  is  challenging  for  the  dearth  of 
experimental  research  on  networks,  the  difficulty  developing  and  managing  the  experiments,  and 
challenges  with  existing  subject  pools  traditionally  used  for  network  experiments.  While 
experimental  research  on  group  processes,  individual  decision  making,  and  cognition  have  long 
histories,  only  recently  have  researchers  returned  to  the  question  of  how  individuals  perform  in 
networks  and  how  the  structure  of  social  networks  affects  individual  and  collective  behavior. 

One  reason  for  this  is  that  network  experiments  require  large  numbers  of  participants  in 
laboratory  setting  which  require  overlapping  but  incongruous  sets  of  individuals  to  interact  with 
one  another.  They  are  thus  particularly  difficult  to  manage.  The  advent  of  methods  for  online 
experimentation  have  greatly  reduced  these  barriers,  but  current  approaches  to  online  network 
research  remain  fragmentary  and  tied  to  problematic  sample  pools. 

Review  of  Network  Experimentation 

How  do  we  think  together?  And  how  do  the  connections  within  a  group  affect  how 
individuals  in  that  group  think  together?  Despite  the  apparent  breadth  of  these  issues,  there  is 
surprisingly  little  research  that  addresses  these  questions.  There  is  a  large  literature  on  groups, 
and  the  detenninants  of  group  performance  (see  review  by  Hackman  and  Katz  2011),  including  a 
significant  amount  on  the  micro  to  macro  connection  between  individuals  and  groups  (Sunstein 
2009).  There  is  a  relatively  small,  if  growing,  literature,  however,  on  group  networks  (see  review 
by  Katz  et  al.  2004),  and  only  a  tiny  subset  of  that  is  about  how  interconnectedness  affects  how 
people  think  together. 

We  define  “thinking  together”  as  the  emergent  nature  of  the  set  of  answers  and 
conceptualizations  of  a  problem  that  confronts  multiple  individuals.  How  to  solve  a  puzzle,  the 
shortest  route  to  work,  what  restaurant  to  have  dinner  at,  are  all  problems  that  have  an  individual 
locus  for  decision  making.  They  are,  however,  collectively  solved,  in  that  we  talk  to,  learn  from, 
and  observe  what  other  people  do.  Our  individual  instincts  in  how  to  solve  a  problem  must  surely 
be  a  mix  of  solving  the  problem  based  on  our  particular  capacities,  while  incorporating  the 
information  available  in  our  network.  The  objective  of  the  proposed  research  is  to  understand,  (1) 
at  the  micro  level,  how  individual  problem  solving  is  affected  by  social  cues,  and  (2)  at  the 
macro  level,  what  are  the  collective,  emergent  properties  of  those  individual  behaviors.  The 
resulting  insights  have  potentially  broad  implications  for  understanding  collective  cognition  at 
various  scales,  from  small  groups  to  societies. 

This  research  builds  on  agent-based  modeling  on  these  issues  by  the  PI  that  focused  on 
one  of  the  key  ways  that  the  network  affects  collective  problem  solving,  which  is  in  balancing 
exploration  and  exploitation  (Lazer  &  Friedman  2007;  March  1991).  Exploration  is  defined  as 
the  development  of  new  solutions  to  a  problem,  and  exploitation  the  use  of  known  solutions.  The 
essential  decision-making  conundrum  is  that  exploration  may  come  at  the  opportunity  cost  of 
taking  advantage  of  current  best  practices.  Lazer  and  Friedman  argue  that  the  structure  of  a 
group's  network  determines  the  balance  between  exploration  and  exploitation.  The  more  efficient 
the  network  of  a  group,  the  more  rapidly  it  converges  on  known  best  solutions.  Convergence  of 
known  best  solutions,  however,  reduces  diversity,  which,  in  turn,  reduces  performance  (Page 
2008)  because  of  the  reduction  in  exploration.  That  is,  efficient  communication  reduces  long  run 
perfonnance. 
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A  key  shortcoming  of  this  research  is  that  it  builds  on  a  very  narrow  body  of 
experimental  research.  This  is  an  approach  to  the  study  of  social  networks  that  largely  fell  into 
disuse  for  many  decades  (Katz  et  al.  2004),  with  a  vigorous  vein  of  experimental  research  on 
small  (usually  <  10)  group  networks  conducted  in  the  1950s  (Bavelas  1950;  Guetzkow  &  Simon 
1955),  but  largely  dying  out  by  the  1960s.  For  the  types  of  questions  we  wish  to  explore,  such  as 
examination  of  the  impact  of  network  topology,  larger  groups  would  be  useful,  which  puts  a 
strain  on  the  standard  human  behavioral  laboratories.  Our  experience  with  traditional  laboratory 
experiments  highlights  the  need  for  an  alternative  model.  In  our  own  experiments,  after  scaling 
down  the  group  size  and  reducing  the  experimental  parameters,  we  still  exhausted  the  entire 
subject  pools  of  the  two  laboratories  we  used,  spent  a  large  amount  of  resources  on  subject  costs, 
and  took  an  entire  year  to  run  trials. 

Online  Experimentation 

In  part  based  on  these  experiences,  and  in  part  due  to  the  emerging  literature  on  online 
experiments,  it  is  clear  that  creating  an  online  group  network  research  laboratory  is  necessary  in 
order  to  pursue  these  research  questions.  Conducting  experiments  online  offers  enormous 
potential  for  studying  the  implications  of  different  network  structure.  Watts  and  collaborators 
(Salganik  et  al.  2006),  Centola  (Centola  2010),  and  Goldstone  (Goldstone  et  al.  2008),  among 
others,  have  broken  critical  ground  in  conducting  experimental  online  research.  However, 
existing  online  experimental  research  uses  customized  software  integrating  with  specific,  pre¬ 
defined  subject  pools  such  as  Amazon’s  Mechanical  Turk  (AMT)  to  perform  single  experiments. 
The  weakness  of  this  decentralized,  ad  hoc  approach  is  a  continued  dependence  on  specific 
subject  pools,  experiments  that  remain  difficult  to  replicate  or  transfer  to  other  researchers,  and  a 
continuous  re-inventing  the  wheel  as  every  social  scientist  must  solve  the  same  programmatic 
issues  like  subject-matching,  consent  management,  and  data  retention. 

The  development  of  online  experiments  and  crowd-based  scientific  research  more 
generally  has  reduced  the  costs  of  research  while  broadening  the  number  of  people  who  can 
participate  and  the  number  of  ways  to  participate.  Current  approaches  to  online  experimentation 
in  the  social  sciences  are  largely  research-specific  and  tied  to  individual  subject  pools  such  as 
workers  on  AMT  and  similar  platforms,  online  samples  such  as  GfK  (formerly,  Knowledge 
Networks),  or  computer-based  experiments  run  in  physical  laboratories.  This  ad  hoc  state  of  the 
field  generates  large  start-up  costs  for  online  experimentation,  makes  the  field  dependent  on 
specific  subject  pools,  and  perpetuates  highly  stylized  experiments  which  are  difficult  to 
replicate  or  borrow  from. 

Other  initiatives  in  online  experimentation  and  citizen  science  demonstrate  the  need  for  a 
more  open,  common  platform  based  on  a  citizen  science  model.  For  large-scale,  online  social 
experiments,  programs  like  Time-sharing  Experiments  in  Social  Sciences  (TESS)  and  Project 
Implicit  have  proven  to  be  powerful  engines  for  collecting  data.  However,  they  provide  limited 
collective  benefits  and  remain  narrow  in  scope.  TESS  provides  a  very  narrow  set  of  experimental 
templates  and  relies  on  a  very  expensive  subject  pool.  Project  Implicit  relies  on  a  volunteer 
sample,  but  is  usable  only  for  a  very  narrow  array  of  scientific  questions.  Citizen  science 
programs  in  other  fields  like  Galaxy  Zoo  and  Fold.it  demonstrate  the  willingness  of  individuals 
to  participate  in  scientific  research  and  the  benefits  of  having  a  publically  open,  participatory 
system.  We  sought  to  combine  the  strengths  of  TESS  and  Project  Implicit  by  creating  systems 
which  facilitate  developing  and  deploying  online  experiments  with  the  strengths  of  citizen 
science  projects  which  generate  communities  of  thousands  of  participants  from  across  the  globe. 
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Summary  of  the  Most  Important  Results 

For  this  proposal,  we  built  the  Volunteer  Science  research  platfonn,  implemented  six 
experimental  paradigms,  and  presented  the  results  of  our  research  to  audiences  at  seven 
conferences  and  wrote  two  papers  to  be  submitted  this  Spring.  Funding  from  ARO  was  used  to 
design  the  Volunteer  Science  website,  develop  the  experimental  Application  Programming 
Interface  (API)  and  documentation,  and  integrate  with  external  resources  like  AMT,  Facebook, 
and  the  survey  software  Qualtrics.  We  also  created  working  versions  of  the  travelling  salesperson 
problem,  prisoners’  dilemma,  public  goods  game,  ultimatum  game,  word  ladders,  and  reaction 
time  experiments.  These  experiments  (along  with  others)  culminated  in  two  papers,  one  focusing 
on  the  design  of  the  platform  and  the  other  on  the  results  of  research  testing  validity  of  the 
platform  and  its  volunteer-based  methods. 

The  Volunteer  Science  Platform 

Volunteer  Science  was  created  to  simplify  design,  make  research  easy  to  share  and  iterate 
on,  and  collectivize  subject  management  processes.  “Volunteer  Science”  is  both  a  website  and 
development  platfonn  for  deploying  web-based  single-user  and  multi-user  behavioral 
experiments.  This  site  is  built  on  top  of  free  open  source  development  tools  including  Django, 
Bootstrap,  HTML5,  and  JavaScript  and  uses  Amazon  Web  Services  (AWS)  to  provide  high- 
perfonnance  and  a  scalable  on-demand  hosting.  These  web-based  technologies  are  agnostic  to 
platform  (Windows,  Mac,  Linux,  Internet  Explorer,  Firefox,  Chrome)  and  also  support  mobile 
devices.  It  is  hosted  on  AWS  which  allows  the  platfonn  to  scale  up  and  down  to  meet  changing 
demand  rather  than  requiring  researchers  to  purchase,  maintain,  and  upgrade  specialized 
hardware. 

The  technical  core  of  Volunteer  Science  for  researchers  is  the  body  of  experiment 
templates  and  API  codes.  The  API  makes  complex  experimental  protocol  like  subject  matching, 
consent,  and  real-time  communication  easy  to  use.  In  developing  fully  functional 
implementations  of  experiments  like  prisoner’s  dilemma  and  the  ultimatum  game,  we  also 
developed  a  template  for  bargaining  games  in  general,  enabling  researchers  to  directly 
implement  other  paradigms  like  the  tragedy  of  the  commons,  public  goods  games,  or  any  Nash 
bargaining  game.  Finally,  included  in  this  API  are  easy-to-use  codes  for  accessing  external 
resources  essential  to  online  experimentation  like  Facebook,  Twitter,  Qualtrics,  and  AMT.  This 
enables  researchers  to  recruit  subjects  broadly  and  access  the  broader  ecology  of  data  available 
online. 

Experiments  and  Research  on  Volunteer  Science 

We  originally  built  Volunteer  Science  to  conduct  a  set  of  social  and  behavioral  science 
experiments  requiring  synchronous  communication  among  subjects  for  researchers  at  three 
universities.  The  model  for  these  experiments  was  to  define  a  meaningful  task  and  have  people 
perform  the  task  within  networks  of  others  performing  the  task.  Thus,  we  built  five  problems 
which  could  be  embedded  in  networks:  the  traveling  salesperson,  prisoner’s  dilemma,  public 
goods,  ultimatum  game,  and  word  ladders.  The  sixth,  reaction  time  experiments,  was  built  to 
expand  the  research  into  broader  disciplines  for  the  purpose  of  validating  the  platfonn. 

Traveling  Salesperson.  We  gave  every  individual  an  identical  Traveling  Salesperson 
Problem  (TSP)  to  solve,  a  canonical  computer  science  problem  that  involves  identifying  the 
shortest  itinerary  for  a  hypothetical  salesman  traveling  among  a  set  of  cities.  We  manipulated  the 
types  of  infonnation  and  connections  to  others’  solutions  and  our  preliminary  finding  confirms 
the  essential  explore/exploit  tradeoff:  more  communication  yielded  fewer  good  answers,  but 
faster  dissemination  of  what  good  answers  were  found. 
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Prisoner’s  Dilemma.  The  classic  bargaining  game  was  embedded  in  networks  of 
individuals  to  determine  whether  or  not  players  in  more  central  positions  in  a  network  would 
have  more  influence  over  the  strategies  of  players. 

Ultimatum  Game.  The  one-sided  bargaining  game  was  created  to  test  the  transitivity  of 
bargaining  to  determine  whether  someone  who  received  a  high  or  low  value  ultimatum  would 
transfer  that  to  a  third  person. 

Public  Goods.  The  public  goods  game  tests  the  ability  for  individuals  to  contribute  to  a 
collective  pool  in  order  to  reap  larger  rewards  while  preventing  free  riding.  In  the  network 
version  we  created,  we  test  whether  or  not  individuals  perfonn  differently  when  they  can  see  the 
contribution  of  their  neighbors  which  may  be  one  other  person  in  the  group,  the  whole  group,  or 
everything  in  between. 

Word  Ladders.  The  word  ladder  game  is  a  puzzle  wherein  individuals  are  given  one 
word  like  “dog”  and,  by  changing  one  letter  at  a  time,  get  the  word  “cat”  (e.g.  dog-cog-cot-cat). 
By  being  exposed  to  others  solving  these  problems,  we  believe  individuals  will  exploit  others 
solutions,  leading  to  more  rapid  convergence  to  the  best  solution,  but  with  fewer  solutions 
generated  than  under  the  non-exposure  condition. 

Reaction  Time  Experiments.  We  developed  versions  of  the  Stroop  (MacLeod  1991)  and 
flanker  tests  (Erikson  1995)  which  detect  very  slight  delays  in  individual  reaction  times  based  on 
simple  manipulations.  Replicating  these  tests  provides  support  for  using  Volunteer  Science  to 
perfonn  time-sensitive  research  and  provided  for  the  template  for  many  other  common 
psychological  experiments  and  measures  pertaining  to  memory,  perception,  and  cognition. 
Scientific  Products 

The  results  of  these  three  years  of  funding  have  been  two  papers  currently  being  written 
and  which  we  plan  to  submit  for  publication  this  Spring.  The  first  paper  documents  the  design  of 
the  Volunteer  Science  system  and  the  advantages  and  disadvantages  of  our  model.  The  purpose 
of  this  paper  is  to  bring  transparency  to  the  system,  document  the  decisions  we  made,  and  offer 
guidance  to  other  researchers  performing  online  experiments  or  working  with  volunteers.  The 
second  paper  will  focus  on  how  we  validated  the  platfonn  by  replicating  a  series  of  canonical 
experiments  and  findings.  We  included  a  broad  variety  of  experiments  and  measures  from 
psychology,  economics,  and  problem  solving  to  provide  robust  tests  of  standard  effects.  The 
results  show  that  our  users  participate  in  good  faith,  are  surprisingly  diverse,  and  can  be  used  to 
study  a  number  of  core  phenomena  of  interest  to  social  scientists. 

Conclusion 

Volunteer  Science  enables  researchers  to  quickly  and  easily  build  online  experiments  by 
giving  them  access  to  existing  templates,  a  range  of  important  features,  diverse  subject  pools, 
and  integrates  directly  with  other  means  of  online  research.  We  have  demonstrated  the  power  of 
the  platfonn  by  implementing  a  range  of  canonical  experiments,  problems,  and  puzzles  and 
tested  how  users  behave  in  social  networks.  And,  now  that  we’ve  validated  the  platform  and  we 
are  starting  to  increase  the  number  of  volunteers  participating  in  our  experiments,  the  results  of 
our  research  should  begin  to  emerge  and  the  platform  should  expand  to  include  a  broader  variety 
of  collaborators  and  experiments. 
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